Laser obstacle detector

ABSTRACT

The present invention provides a device for assisting a person in determining the presence or identity of obstacles in the person&#39;s path, characterized by a housing, a laser projector for generating a laser pattern on a surface in said person&#39;s path, a receiver for at least receiving a plurality of images of the laser pattern reflected from the surface and for generating a signal corresponding to said laser pattern reflection and a processor for processing said signal to at least determine the presence or identity of an object, and a warning generator for generating a warning to the person. The present invention also provides a method for assisting a person in determining the presence of an object in a person&#39;s path comprising generating a laser pattern comprising a plurality of laser lines on a surface, distinguishing between one or more straight line segments in the laser pattern and distorted line segments of said laser pattern after they are reflected from said surface and evaluating the distortions in said line segments of said laser pattern to determine the presence or identity of an object in the person&#39;s path. The present invention can be used to assist the visually impaired in seeing objects in their path or can be used by a sighted user in game play or at night for detecting obstacles. Some of the functions of the present device can be provided in a smart phone which can be adapted with a laser projection as a separate attachment or as part of the smart phone circuitry.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/233,563 filed Apr. 23, 2014, which is a 371 U.S. National Phase ofPCT/US2012/47547 filed Jul. 20, 2012, and claims the benefit of priorityto U.S. Provisional Application Ser. No. 61/509,788, filed on Jul. 20,2011, which are herein incorporated by reference.

BACKGROUND

For the blind or visually impaired, the current state of technology,besides guide dogs, includes alternative devices based on mechanicaldesigns (i.e. various types of canes and guide mechanisms) orultra-sonic technologies. Although the blind and visually impaired havemade great strides in education, employment, and information access,they have not benefited from any real breakthroughs in practical andaffordable mobility devices that facilitate safe and independent traveloutside of the home.

Although users may become very adept with a “pure” mechanical device,such as the traditional “long cane” or “white cane,” there are inherentlimitations to its use, most notably the requirement that the deviceactually touch the obstacle by a “swipe” or a “thrusting” or “probing”technique. This fact can put the person in considerable danger, even ifsuccessfully employed since typically, the probing motions are limitedto close, ground-level obstacles.

Some recent attempts to improve upon the use of canes, involve the useof ultrasound. See, for example, U.S. Pat. No. 6,469,956, hereinincorporated by reference. Scanning with an ultrasound wide beam willnot easily find openings, especially from a distance. The device forgenerating an ultrasound beam is conical in shape, about 48 inches indiameter at 14 feet. Wind and temperature changes affect the ultrasoundbeam and will give false distances and alarms. If approaching anobstacle at less than 45° degrees, the ultra sonic signal tends to missthe obstacle because it is reflected away from smooth surfaces (e.g.,smooth walls, and some smooth, contoured obstacles) and is a knowndisadvantage of ultra sonic detection devices in general. Thus, thetypical ultra-sonic devices that have been marketed are simply notreliable enough to inspire the trust necessary for a blind person torely on the sensory signals (i.e. sounds, musical notes, etc.) therebynegating one of the major positives to be expected from a primary visualassistance device, namely, a growing sense of self-confidence and trustin the device.

Use of guide or companion dogs has provided some people with increasedmobility and self-sufficiency, but this benefit is limited to a smallportion of the overall potential patient population for many reasons.Although guide dogs enjoy tremendous support among the general public aswell as among the visually impaired population, there are a significantnumber of people who cannot use a guide dog or who cannot cope with someof the disadvantages inherent in using a “sight instrument” withindependent action capabilities. Many people are allergic to, or don'tlike, dogs or their feed/grooming aids, and caring for the dog presentsstresses and challenges that many vision and mobility impaired peopleare not able to handle without constant assistance from a sightedperson. Dogs, just like people, have personalities which can bedifficult to “match” and they also get sick, infirm and die, creatingserious emotional and practical challenges.

An object of this invention is to provide an effective obstacledetection accessory to smart mobile electronic systems, such as portablegame systems and mobile phones, for assisting the visually impaired atlow cost.

SUMMARY OF THE INVENTION

In a first embodiment of the present invention, a device for assisting aperson in determining the presence or identity of obstacles in thatperson's path is provided. The device includes a housing; a laserprojector for generating a laser pattern on a surface in said person'spath; a receiver for at least receiving a plurality of images of saidlaser pattern reflected from said surface and for generating a signalcorresponding to said laser pattern; a processor for processing saidsignal to at least determine the presence or identity of an object; anda warning generator for generating a warning to said person.

In a further embodiment, the invention provides a means toelectronically detect obstacles comprising: either a self-contained,independent hand-held device or a housing that can mounted to a vehicle,walking cane, or other aid; a laser projector that generates a pluralityof non-parallel, intersecting laser lines in the form of a pattern; asolid state camera that records the reflected laser line patterns asimages; an image background removal processing means to reduce computingtime and the need for powerful computers; an edge detection processingmeans; a processing means to determine if the laser pattern projected ata non-normal angle to an object is recorded by the camera as adistortion of the original pattern; an annunciation means, audible ortactile, to alert the user of an impending obstacle. In addition, withthe storing of a plurality of distortion patterns in solid state memory,the invention could also be employed to identify obstacles by comparingthe camera's recorded image against a plurality of stored images.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate preferred embodiments of theinvention as well as other information pertinent to the disclosure, inwhich:

FIG. 1 is a front perspective view showing a person holding the deviceof this invention and approaching a stairway, further illustrating anon-uniformity of the projected laser pattern;

FIG. 2 is a side elevation view of a cane equipped with the device ofthe present invention;

FIG. 3 is a cross-sectional view of a preferred detector system;

FIG. 4 A-C comprise reflective laser patterns in a “X” configurationwith additional laser lines shown in FIGS. 4 B and 4 C;

FIG. 5 is a reflective laser pattern in a “X” design projected on asurface other than normal)(90°);

FIG. 6 is a front perspective view of the laser pattern of FIG. 5 beforeit encounters a stair;

FIG. 7 is a front perspective view of the reflective laser pattern ofFIG. 5 after it strikes the stairs of FIG. 6, further showing distortedand broken straight line segments;

FIG. 8 is front perspective view of the stairs and floor captured as animage with the laser module pulsed off;

FIG. 9 is a front perspective view showing a negative image of thestairs and floor of FIG. 8 which can be used to subtract the backgroundfrom the laser image of FIG. 7;

FIG. 10 is an isolated laser line pattern after subtracting thebackground image;

FIG. 11 is a front perspective view of a smart phone equipped with alaser projector attachment and a reverse perspective view of said smartphone showing phone accessory connector points;

FIG. 12 is a front perspective view showing a smart phone embodiment ofthe present invention using its built in camera to record a laserpattern on a stairway in the person's path; and

FIG. 13 illustrates a front perspective view of the smart phoneembodiment of the present invention and stairway, illustrating how anobstacle at night may appear on said phone's screen to a sighted userwhen used in a game or at night for detecting obstacles.

DETAILED DESCRIPTION

Although obstacle detection can be used in a variety of embodimentsincluding human hand-held games, vehicles, wheelchairs, robots, and anumber of moving objects, one preferred embodiment is shown in FIG. 1 asa walking cane for the blind and visually impaired.

Referring to FIG. 1, a blind or visually impaired person 10 carries acane 11 as he walks. The cane typically is constructed from light weightmetal, fiberglass or plastic with a tip 17 at the bottom end to help theperson “feel” his way along. Attached to the top side of the cane 11,detector system 12 consists of an infrared laser projector 14 and solidstate camera assembly 16. Other elements of detector system 12 arereferenced in the discussion relevant to FIG. 3. The infrared lightspectrum is used for eye safety, unobtrusiveness and for easierdetection by a solid state camera imager. An infrared laser beam pattern18 is projected from the projector 14 in front of the person 10. Thelaser pattern is aligned such that it extends approximately twenty fourto thirty inches in front of the cane. Stairs 24 represents an obstacleoften encountered by visually impaired people in the act of walking.Other obstacles could include, street curbs, pot holes, other people,vehicles, walls, etc.

In this embodiment, the laser pattern projected at an angle onto asurface consists of a plurality of non-parallel, intersecting laserlines arranged in an “X” pattern 18. Other non-parallel, intersectingline patterns can be used as well and will be discussed later. On a flatsurface the legs 19 and 20 of pattern 18 appear as an “X”, but appear tobreak into several segments 21 and 22 when projected onto stairs 24 atan angle. One important aspect of this invention is the ability of thedetector system 12 to distinguish between continuous straight linesegments and those same line segments appearing as distorted or brokenup into one or more non-continuous angled line segments.

In FIG. 2, a typical walking cane 11 can be equipped with a detectorsystem 12 attached to the cane. A top side attachment is preferred inorder to project a laser pattern from laser projector 14 about twentyfour to thirty inches in front of the cane tip 17 and for the camera 16to have an unobstructed view of the projected pattern. The detectorsystem 12 can be attached to cane 11 in a variety of ways including hoseclamps, screws, clip-on, snap fit and adhesives.

Referring to FIG. 3, a preferred embodiment for the detector system 12consists of a case enclosure or housing 41, a solid state laserprojector 14, a receiver or camera assembly 16, a printed circuit boardassembly 58 and a speaker 45. The laser projector 14 consists of a laserdiode module 44, a set of cylindrical lenses 46 and 48 oriented at 90degrees to each other and a cable 43 attached to printed circuit board58. While cylindrical lenses are shown, alternatives includingdiffraction optical elements can be employed. Diffractive elements arethin phase elements that operate by means of interference anddiffraction to produce arbitrary distributions of light. Thesediffractive elements in the form of analog, continuous-phase diffractiveelements represent cost-effective solutions and enable the constructionof more general laser patterns.

Further in FIG. 3, a receiver, such as a device that can measurereflected electromagnetic wave patterns, such as a solid state cameraassembly 16, consists of a body 54, a focusing lens 56, a twodimensional imager and board with electronics 50 and a cable 51connected to a main circuit board 58. The imager can be fabricated fromCMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge CoupleDevice) technologies and is commonly found in web cams and cell phonecameras. The resolution of the camera used in this embodiment can betypically 1 or 2 Mega Pixels. But as technology advances and costs comedown, higher resolutions can be employed in meeting the objectives ofthis invention.

A preferred piece of the detector system 12 is a printed circuit board58 containing electronic processing capabilities using a microprocessor,memory, input/output facilities, connectors and support for the firmwareused to control the laser pattern, camera functions and image processingalgorithms used to evaluate distortions in the beam pattern. Anotherpreferred aspect of this invention is the electronic detector's ability,in real time, to discern changes in the beam pattern as an indicator ofa pending object in the user's path. The fabrication of the printedcircuit board and its electronic components is widely known to thoseversed in the art and will not be explained in detail here.

The remaining piece of the detector system 12 is an annunciator means.In this embodiment, it consists of a speaker 45 and a cable 47 connectedto printed circuit board 58 used to sound an alarm or issue a voicemessage. However this could be a jack for head phones, vibrating motor,connection to finger actuators to represent Braille or any combinationthereof.

Typical laser line patterns are represented in FIGS. 4a, 4b and 4c . Byextension, any number of non-parallel, intersecting projection lines canbe fabricated using diffractive elements. In FIG. 4a , an “X” pattern 60consisting of line segments 62 and 64 form a projected pattern normal toa flat surface. FIGS. 4b and 4c show alternative forms of line segments.Any one of an infinite set of patterns could be used.

By using a non-parallel intersecting line pattern, the orientation ofthe invention whether up or down (pitch), rotated (yaw) or turned (roll)relative to an obstacle's position becomes less critical to thedetection of distorted or broken line segments in the projectionpattern. This is because one or both of the beam segments will appear,in the majority of orientations, as non-continuous, broken segments. Itfollows that increasing the number of non-parallel intersection linesegments as in FIGS. 4b and 4c may further improve the inventions'sorientation tolerance.

In FIG. 5, it can be seen that when an “X” pattern 60 is projected ontoflat surface, but at an angle other than normal (90 degree), the linesegments 62′ and 64′ and their intersection 61 are no longer symmetricalto each other. However, this non-symmetry is of little consequence inachieving the objectives of this invention. As will be seen,discontinuity, distortion or line segments broken into multiple angledline segments are the important indicators of a potential obstacle.

FIG. 6 consists of stairs 66 rising above a flat floor 68. Stairs aretypical obstacles that blind or visually impaired people must contendwith and are used here as an example. Other obstacles like pot holes,street curbs, vehicles, other people etc. can be detected by theinvention in a similar manner. In FIG. 6, the laser line pattern 60,composed of straight lines 62′ and 64′ and their intersection 61 isnon-symmetrical but consists of straight, continuous line segments.Consequently, this pattern is processed by the detector system 12 inFIG. 3 as a normal pattern and not an obstacle.

As seen in FIG. 7, when the pattern 60 in FIG. 6 strikes the stairs atan angle, it now appears visually distorted and broken into severalnon-continuous straight line segments. From the recording camera's 16angled point of view, the resulting projected laser line pattern 70 inFIG. 7 consists of non-continuous line segments 74 and 82 intersectingat point 61. Subsequently, non-continuous line segment 74 is composed ofindividual line segments 76, 78, 80 and 81. Non-continuous line segment82 is composed of individual line segments 84, 86, 88 and 90. When astraight laser line is projected at a non-normal angle onto multiplesurfaces of different heights, a three dimensional optical effectoccurs. Here, normally projected continuous lines appear broken intoseveral non-continuous line segments. Using this information, embeddedprocessing firmware can detect the presence of an obstacle.

An object of this invention is to simplify laser line pattern detectionwhile accurately ascertaining the presence of an obstacle. In mostcomputer vision systems, a significant amount of computer processingpower is expended on isolating the desired image pattern from thebackground. In this invention the laser diode module 44 in FIG. 3 ispulsed on-off at a rate of approximately 10 to 30 times a second with anapproximate 50% duty cycle. The camera assembly 16 in FIG. 3 isprogrammed to capture a received image while the laser module is pulsedon and again while it is pulsed off In FIG. 7, the background stairs,floor and projected laser pattern are captured as an image while thelaser module is pulsed on. In FIG. 8, only the stairs 66 and floor 69are captured as a second image when the laser module is pulsed off. Themicroprocessor firmware on board 58 of FIG. 3 then digitally inverts theimage in FIG. 8 resulting in a negative image in FIG. 9. The negativeimage in FIG. 9 is then digitally added to the image in FIG. 7,effectively subtracting the background from the image in FIG. 7. Theresult is an isolated laser line pattern in FIG. 10. Through thefirmware processing, distorted or non-continuous line segments aredetermined to be an obstacle.

Digital inversion is preferably accomplished at the pixel level. Since adigital image comprises a multitude of pixels forming a two dimensionalbit map, each pixel has an X and Y coordinate position in the bit map,an intensity value Z and a color value W. This information is in theform of a binary representation. As a simple example, if the originalpixel of the background image is brown with a component color value of 01 0 1 1 1 0 0, its binary compliment would be 1 0 1 0 0 0 1 1. By addingthe original value and its compliment together, the result is 1 1 1 1 11 1 1 binary. In most systems this is white and the new background atthat pixel is now white. By repeating this process for each pixel in theX, Y coordinate space, the entire background can be erased and replacedwith a white background.

Now, when the laser module is pulsed on, the pixel located on the laserline may have a color other than brown. When the laser line pixel colorvalue is added to complimented background pixel color value storedearlier when the laser module was pulsed off, than the resulting pixelwill appear as a non-white color on a white background. For example, ared pixel with a component color 1 1 1 0 1 1 1 0 when added to thecomplimented background color of 1 0 1 0 0 0 1 1, results in a colorvalue of 1 0 0 1 0 0 0 1 with a carry bit. Since this value is otherthan white (1 1 1 1 1 1 1 1) we can say that pixel is part of the laserline.

Where the color values of the complimented background pixel and theforeground laser pixel are too close to the background color, we caninstead use the intensity value Z of the pixels to determine if thepixel is part of the laser line. If that also fails, we can drop thatpixel and by extrapolation from the surrounding laser line pixels, fillin the questionable pixel. As discussed previously, the main objectiveis to determine whether the laser line recorded by the camera appearsdistorted or no longer continuous and therefore infer that an obstacleis present.

Additional means to enhance the received laser line pattern image whileeffectively removing the image background may be added as well. Theseinclude but not limited to: 1} adding a narrow band optical infraredfilter corresponding to the laser module wavelength to the cameraassembly 16 in FIG. 3., 2) voltage/current modulation of the lasermodule to create a series of laser patterns with varying intensities tofilter out ambient sun light, and 3) manipulation of the contrast,brightness, color saturation and exposure controls in the camera'simager firmware to enhance the received laser light pattern and reducethe effect of the background.

Once the image background is effectively removed, a means of edgedetection is employed to determine if a normally straight, continuouslaser line appears broken into one or more non-continuous, angled linesegments. Digital image processing employing forms of edge detection incomputer vision systems is well known in the art. While the preferredembodiment of this invention uses a detection method called Cannyoptimal edge detection, developed by John F. Canny in 1986, many othertypes of detection methods can be employed such as Gaussian Function,Gaussian Convolution, Gaussian Smoothing, DoG (Difference of Gaussianmask), Zero-crossings, Grey-Level Gradient, and others. Canny isparticularly useful in this invention because it combines GaussianSmoothing with thresholds of pixels relative to adjacent pixels. Whenthe X, Y coordinates of these high threshold pixels are calculated inthe processing firmware, a line or line segment can be determined Bycomparing the continuity of the original laser beam pattern linesegments with the received image of non-continuous, broken linesegments, the processing firmware can make a go, no-go obstacledetection decision.

While an object of this invention is obstacle detection, furtherprocessing of the received line pattern can also lend itself to obstaclerecognition. As opposed to detection alone, recognition can determinewhat kind of obstacle is being observed. Comparison between receivedpatterns of stairs, pot holes and walls against pre-determined patternsof these obstacles makes this possible. Once a definitive decision ismade by the firmware that an obstacle exists and what the obstacle is,feedback through an alarm sound, voice message, vibration or mechanicalactuation can instruct the user to what kind of obstacle is ahead.

Another embodiment for guiding the visually impaired or for gamesconsists of a laser projector attached to a smart phone. Using the sametechniques described earlier in projecting a laser pattern, recordingthe pattern as an image against a background scene, digitally removingthe background scene, detecting non-uniformity of the projected patternline segments signifying an obstacle and communicating that to the user,a smart phone already contains most of the elements required for theinvention; namely a) a solid state camera, b) processor, c) speaker orearphone, d) vibration device, and e) a portable housing. The only itemmissing is a laser pattern projector. In this invention the laserprojector consists of an add-on laser pattern projector accessoryattached to the smart phone and digitally connected through the smartphone's USB or synchronizing connector through a snap-on back case 104in FIG. 11.

By using an existing smart phone hardware platform and its ability torun firmware applications, an extremely cost effective obstacle detectoris achieved with just the addition of a low cost, solid state, laserpattern generator. When playing a game such as “manhunt” in the dark theinvention would help players avoid stepping into holes and running intotrees or other obstacles. It could also be employed as a critical partof a scavenger hunt game where players use the invention to detect andidentify objects in low light or dark spaces.

Another aspect of the invention when used with a smart phone is itssubtlety. The visually impaired do not want to stand out in a crowd.Using a smart phone to scan for obstacles allows the visually impaireduser to blend in with many other smart phone users as they read messagesor use applications while walking.

In FIG. 11, a possible mounting for the laser projector is in the formof a smart phone back case is illustrated. The outside of the back case98 houses the laser projector assembly 100. The electrical connectionsfrom the laser projector 100 run along the inside of case 102 to thephone accessory connector points 104 at either the bottom or along theside depending upon the smart phone configuration. In this way, theapplications firmware in the smart phone can be written to pulse thelaser through the phone's accessory connector to facilitate thebackground scene removal process discussed earlier. The case alsofacilitates a convenient and robust method for mounting the laserprojector assembly 100.

Referring to FIG. 12, smart phone 98 is held by the user so that itsbuilt-in camera 106 points out and in front. As a snap-on back caseaccessory, the laser pattern projector 100 projects a pattern 18 ontoany obstacle that may be 24″ to 30″ in front of the user. In thisexample, the projected pattern onto stairs 24 appears as a distortedimage to the phone's camera. As described earlier, the snap-on laserprojector in a smart phone accessory case provides both a fixed locationand a means to connect to the phone's digital accessory signals at thebottom or on the side. Just as in the laser cane embodiment, the phone'sbuilt-in camera and processor pulses the laser pattern projector,records the scene with and without the laser pattern, digitallysubtracts the scene's background to isolate the distorted patternedreflected form the stairs, applies the edge detection algorithm,determines if the reflected pattern is an obstacle or not and throughthe phone's built-in speaker, earphones and/or vibrator communicates tothe user that an obstacle is ahead.

FIG. 13 illustrates how an obstacle at night may appear on the phone'sscreen to a sighted user when used in a game or at night for detectingobstacles. Phone 98 with its built-in screen 108 can display thecaptured, distorted image 110 of a potential object or obstacle.Together with sound and/or vibration, the screen could also indicate animpending object and alert the user.

In addition to detecting obstacles, predetermined distortion patternsstored in the processor's memory could also be used to identify objectsor obstacles. The application firmware in the phone would storerepresentative patterns of pot holes, curbs, walls, stairs, etc. andcompare the stored patterns against the non-uniform patterns recorded bythe phone camera. Once an approximate match was made, communicationthrough the phone's speaker, earphone, screen text or coded vibrationsto the user could alert the user to an obstacle and it's type. The usercould therefore take appropriate action to avoid injury or damage.

The present invention will be further explained and illustrated by theattached claims.

What is claimed:
 1. A device for assisting a person in determining thepresence or identity of obstacles in that person's path, characterizedby: a housing 41; a laser projector 14 for generating a laser pattern ona surface in said person's path; a receiver 16 for at least receiving aplurality of images of said laser pattern reflected from said surfaceand for generating a signal corresponding to said laser pattern; aprocessor 58 for processing said signal to at least determine thepresence or identity of an object; and a warning generator 45 forgenerating a warning to said person.
 2. The device of claim 1, whereinsaid laser projector comprises an infrared laser projector.
 3. Thedevice of claim 1, wherein said receiver comprises a camera.
 4. Thedevice of claim 1, wherein said laser pattern comprises a plurality ofnon-parallel laser lines.
 5. The device of claim 4, wherein saidplurality of non-parallel laser lines are arranged in an intersectingpattern
 6. The device of claim 1, wherein said processor can distinguishbetween continuous straight line segments generated by said laserprojector, and distorted versions of said continuous straight linesegments.
 7. The device of claim 6, wherein said distorted versions ofsaid continuous straight line segments represent an indication of anobject in that person's path.
 8. The device of claim 1, wherein saidreceiver, processor, and warning generator comprise portions of a smartphone, and said laser projector comprises an attachment to said smartphone.
 9. The device of claim 8, wherein said receiver comprises abuilt-in camera of said smart phone and said built-in camera recordssaid person's path with and without the laser pattern, and digitallysubtracts a background scene from said person's path to isolate saidlaser pattern reflected from its surface.
 10. A device for assisting aperson to determine the presence or identity of an object in thatperson's path, characterized by: a smart phone 98 having a camera 106and processor and warning generator for generating a signal which can befelt or heard by said person; a laser projector attachment 100 forelectrically connecting to said smart phone 98, said laser projectorattachment 100 providing a laser pattern 18 which can be recorded by thecamera 106 and then processed by the processor to detect anon-uniformity 110 of the projected laser pattern indicative of anobject in said person's path.
 11. A method for assisting a person indetermining the presence of an object in that person's path,characterized by: generating a laser pattern comprising a plurality oflaser lines on a surface in said person's path; distinguishing betweenone or more straight line segments in said laser pattern and distortedline segments of said laser pattern after it is reflected from saidsurface; and evaluating the distortions in said line segments of saidlaser pattern to determine the presence or identity of an object in saidperson's path.
 12. The method of claim 11, wherein said evaluation stepcomprises recording said reflected laser pattern as an image against abackground scene and then digitally removing the background scene. 13.The method of claim 12, wherein said distinguishing step comprisesdetecting a non-uniformity in one or more laser pattern line segmentsindicative of an obstacle in said person's path.
 14. The method of claim11, further comprising applying an edge detection algorithm to a storedimage to determine if the reflected pattern is an obstacle.
 15. Themethod of claim 14, wherein said method further comprises generating awarning signal to said person when an obstacle is detected.
 16. Themethod of claim 14, wherein said method comprises generating a warningsignal to said person that helps said person identify said object.